@yandongxiao @kevincai plz help review

don't try to use FE_OBSERVER_NUMBER to hack the current fe_entry_point.sh

It is so dangerous that a single improper env var update can ruin the entire cluster.

don't try to use FE_OBSERVER_NUMBER to hack the current fe_entry_point.sh

It is so dangerous that a single improper env var update can ruin the entire cluster.

Got it. In that case, we may need to add a new FE observer component.

This is much needed, thank you!

This is much needed, thank you!

I still need some time to test this feature. Thanks!

@kevincai @yandongxiao plz help review. Thanks!

Thanks for the contribution. The overall mechanism — a dedicated observer workload plus the minimal fe_entrypoint.sh change in StarRocks/starrocks#67778 — looks like the right direction. However, I have four concerns. The first one is blocking.

1. Observer pods have no resolvable FQDN, and no Service routes to them (blocking)

MakeStatefulset initially sets ServiceName to <cluster>-fe-observer-search (the new case in service.SearchServiceName), but SyncCluster immediately overrides it:

expectSts.Spec.ServiceName = feSearchServiceName(src.Name) // "<cluster>-fe-search"

That service does exist (created by the FE controller), but its selector is built from load.Selector(src.Name, feSpec), i.e. {app.starrocks.ownerreference/name: <cluster>-fe, app.kubernetes.io/component: fe}, while observer pods carry {<cluster>-fe-observer, fe-observer}. The selector never matches, so observer pods never appear in the Endpoints of <cluster>-fe-search, and cluster DNS never publishes A records for <pod>.<cluster>-fe-search.<ns>.svc.cluster.local.

The pod can still resolve its own FQDN locally (kubelet writes hostname/subdomain into /etc/hosts), so ALTER SYSTEM ADD OBSERVER "$MYSELF:$EDIT_LOG_PORT" is actually sent — but the FE leader cannot resolve that FQDN, BDBJE cannot connect back, and the entrypoint times out and exits → CrashLoopBackOff. Since the operator always sets HOST_TYPE=FQDN, the feature cannot work as currently implemented.

Additionally, because the dedicated observer services were removed, no Service selects observer pods at all — there is no entry point for clients to reach observers, which seems at odds with the stated goal of query load balancing.

Suggestion: keep a dedicated headless search service for the observer StatefulSet (selector = observer labels), which also makes the currently-dead *StarRocksFeObserverSpec case in service.SearchServiceName meaningful again. It is also worth clarifying how clients are expected to reach observers (e.g. a dedicated external service).

2. deleteLegacyObserverServices should be removed

It deletes <cluster>-fe-observer-search / <cluster>-fe-observer-service, which were only ever created by earlier revisions of this PR and never released — no real deployment can have these resources.

It also runs on the enabled path on every reconcile, and clearObserverResources runs on the disabled path (the default) on every reconcile of every cluster. DeleteService/DeleteStatefulset log "delete service/statefulset from kubernetes" before the existence check, so every reconcile of every cluster emits misleading delete logs plus pointless GETs. Consider gating the disabled-path cleanup on src.Status.StarRocksFeObserverStatus != nil.

(Minor: the !apierrors.IsNotFound(err) checks are redundant — DeleteService/DeleteStatefulset already return nil on NotFound.)

3. No deregistration on scale-down / disable

Neither the operator nor fe_entrypoint.sh ever runs ALTER SYSTEM DROP OBSERVER. Scaling observerNumber down or setting enabled: false leaves permanent Alive=false entries in SHOW FRONTENDS. This matches the existing FE follower convention (manual DROP), but observers are precisely the role intended to be scaled dynamically, so dead entries will accumulate much faster. This should at least be documented as a known limitation, with manual cleanup instructions.

4. Old image + observerSpec.enabled=true silently adds FOLLOWERs (dangerous)

IS_FE_OBSERVER is only understood by fe_entrypoint.sh after StarRocks/starrocks#67778. With any older FE image, the env var is ignored and the script falls through to ALTER SYSTEM ADD FOLLOWER — the observer replicas join as voting followers and silently change the BDBJE quorum. This is exactly the failure mode raised earlier in this thread regarding env-var-driven behavior.

Suggestion: add an operator-side image version guard (IsLowerThanAny is already used this way for the BE preStop path) and document the minimum required FE image version for observerSpec.

@yandongxiao Thanks for your comments. I will make some improvements.

@yandongxiao plz help review. Thanks!

@yandongxiao can you please review these changes i'm also interested in getting the capability for our starrocks deployments.